Image processing involves the use of multiple images of a scene to generate a new image. The images input to image processing may include video images. For example, stereographic imaging generates three-dimensional (3D) images from two images of the same scene. The images may be generated using two cameras, with each camera recording the scene from different viewpoints. In some cases, the two cameras may record the scene in different colors. In other words, some of the same elements of a scene may appear differently colored to each of the cameras. This may be due to a difference in lighting and reflection when viewing a scene at different viewpoints. It may also be due to other factors, one of those factors being characteristics of the cameras such as focal length, aperture, color balance, automatic luminance control. The images used may be video images.
Typically, image processing involves geometric processing, and photometric and color processing. Geometric processing usually extracts corresponding feature points from two images. A feature point is a specific point in an image that has been detected and that is characterized by specific image signal characteristics such as color, spatial frequency or edges. Corresponding feature points are two or more feature points in two or more images that have been identified to correspond to the same semantic scene element. Geometric process also calculates one or more of: camera position, internal camera parameters, scene depth, and disparity of objects. The disparity is the deformation of image content between two images e.g. caused by the 3D shape of objects. Extraction of corresponding feature points is typically performed in two steps: detecting feature points, and searching for corresponding feature points.
One method for feature point detection and correspondence is called Scale Invariant Feature Transform (SIFT). The SIFT method employs a detector based on Difference of Gaussian (DoG) and a descriptor that relies on the distribution of gradient vectors orientation.
Photometric and color processing deals with color differences between images. Typically, color differences are not an issue in image processing if they result from an intended operation. For example, in U.S. Pat. No. 5,583,665 to Gregory, Jr., et al., an image is combined with a color transform, called a “color tracer” or “color grid.” The color transform is based on a Color Look Up Table (CLUT). The CLUT entries form a grid in color space. In some cases, a CLUT may be defined just for a local image region. The grid may contain colors that are not in the image. The CLUT may also be in the same file as the image. Typically, the CLUT is empty at the beginning of image processing. However, once an operator edits the color in the image, the edits may be registered into the CLUT. The CLUT can then be applied to other images using interpolation.
Photometric and color processing usually involves the extraction of corresponding colors in two or more input images. If these colors are arranged into a table, the table can be used as a CLUT in order to estimate one of: color differences, object reflectance, illuminant characteristics, light distribution, photometric camera parameters, and appearance of color in the images.
In U.S. Pat. No. 5,212,546 to Arazi, et al., a manual method for obtaining corresponding colors is described. Corresponding colors are organized into a CLUT. An operator determines and adjusts the CLUT by applying it to an image and comparing the transformed image to a reference image.
Another manual method for obtaining corresponding colors is described in the U.S. Pat. No. 6,137,903 to Dichter. This method results in one pair of corresponding colors. An indication is made on a computer screen of a group of source colors in a source image. A corresponding group of reference colors in a reference image is also indicated. The source colors and the reference colors are averaged resulting in a representative source color and a representative reference color, respectively.
Photometric and color processing also typically includes color mapping. Color mapping involves the use of known color correspondences between source colors in a source image and reference colors in a reference image. In U.S. Pat. No. 6,137,903 to Dichter, a color mapping method is based on a single color correspondence. Source colors are mapped to target colors by one of the following two steps. If a source color is the same as a representative source color, the source color is mapped to the representative reference color. In all other cases, the difference between the source color and the representative source color is calculated. The source color is mapped to a target color that has a corresponding distance to the representative reference color.
Another color mapping method is proposed in U.S. Patent Application Publication No. 2009/0257652 by Lie, et al. The method takes, as input, source images and a reference image (or portions of such images). Color correspondences are assumed to be available. In other words, a list of corresponding source and reference colors are known. The colors are assumed to sufficiently populate the color space. The method calculates a two-stage color transform capable of transforming the colors of the source image into colors of the reference image. The first stage calculates one-dimensional look up tables (LUTs) for each color channel. Color channels can be typically R, G, B for red, green and blue, respectively. If, for a single source color, multiple reference colors are observed, a mean reference color is calculated. The obtained tables can be smoothed by polynomial fitting. As an alternative, the 1D LUTs can be obtained by matching the cumulative distribution functions. In this case, color correspondences are not needed. The second stage transforms intermediate colors to reference colors. The intermediate colors are the source colors transformed by the first stage. The second stage allows cross color channel processing and calculates a linear 3×3 matrix by regression on the color correspondences.
Known methods for the extraction of corresponding colors in two or more input images include direct and indirect approaches. An example for a direct approach is the Maximally Stable Color Regions method. In this method, uniformly colored regions (“color blobs”) in the input images are selected that are stable against changing background. The correspondences between colored regions in the selected regions are then established.
In indirect approaches for the extraction of corresponding colors, colors are often chosen to be the colors of corresponding feature points. The corresponding features may be detected using the same methods as used in geometric processing. Those feature points are assumed to be robust against geometric changes. For example, corresponding feature points in two stereo images may be extracted using the SIFT method. Correspondences of the near neighborhood of a feature point in one input image may be established with the near neighborhood of a corresponding feature point in the second input image. Further, corresponding feature points can be exploited when color differences are to be compensated in more than two input images.